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Background of the Invention 
Field of the Invention 

The present invention relates to a data process 
technology for processing data from a sensor used in 
a robot, etc, and a configuration technology for a 
language process interface used for providing 
instructions to a robot or a device, etc. from a 
user. In particular, the present invention relates to 
a method and apparatus for processing data with words 
(data to which words are attached), which provide the 
classification and storage methods for the sensor 
data and the usage method for the stored data for 
easily assessing the circumstances based on the 
sensor data, and further provide means for linking 
the human language to a behavior and operation of a 
robot, a data storing method thereof, and the usage 
method for the stored data. 

Description of the Related Art 

In a general sensor data processing method, a 



certain characteristic is extracted by performing a 
numerical process on the sensor data, and a next 
process method is determined or modified according to 
the characteristic. At this time, although a certain 
condition is assumed based on the characteristic 
value obtained from the sensor data and the next 
process method is determined, the condition is 
assumed only by a predetermined procedure because the 
characteristic, the value, and the classification 
expected in the next process are all predetermined. 

Since a sensor used in a robot, etc. is a device 
for measuring a certain state or a change of this 
state, it is desirable to be able to correctly judge 
the status and state of the location from the 
obtained sensor data. However, since, generally 
speaking, the amount of data from a sensor is 
enormous, and in most cases the sensor data may 
include errors due to noise, etc., it is very 
difficult to make a simple status judgement based on 
a list of numerical values of the sensor data. 

On the other hand, generally speaking in a 
command provided for operating a device such as a 
robot, etc., the speed or the location of the robot 
is directly designated by a numerical value, such as 
a command to operate at a speed of lm/sec, to rotate 



at Irad./sec, or to move to a location (100, 200). 
In electric home appliances such as an air 
conditioner, etc., there are only such setting 
commands as for setting a temperature to 20 degrees, 
setting the air conditioner to a "dry mode", "sleep 
mode", etc* based on predetermined setting values. 

However, when one man gives an instruction to 
another man, an instruction is not given like this. 
An instruction which is given from one man to another 
is usually an abstract instruction, simply like 
"Walk!" or "Walk faster!", in which the speed or the 
angle of the legs are not specified. In electric home 
appliances such as an air conditioner, etc. saying 
"Higher temperature!" or "Cooler! 11 is more natural 
than designating a numerical value such as 20 
degrees . 

If an instruction can be given to devices such 
as a robot, air conditioner, etc. using these 
expressions, it can be expected to provide a more 
natural interface between a man and a machine. 

In addition, there is a problem in the case 
where an instruction is given to a machine using 
natural words like these, in that an instruction to 
increase a temperature varies depending on the person 
issuing the command. For example, one person may say 
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"Higher temperature!", but another person may say 
"Raise the temperature!". "Warmer!" can also be used 
with the same meaning. 

Conventionally, since in order to solve this 
problem a variety of types of instructions are 
anticipated, listed, and stored, configuring a system 
for understanding commands like these requires a lot 
of work and an enormous memory capacity, and 
therefore is not practical. 

In other words, conventionally, since there is 
no means for learning and storing the specific 
meanings of the commands "Warmer!", "Walk!", etc., a 
system which can flexibly respond to these commands 
has not been realized. 

As described above, although a certain kind of 
state and a change of the state can be determined 
based on the sensor data, it is difficult for men to 
judge the condition of a target object by observing 
the sensor data, because the sensor data are not 
appropriately classified. Since a characteristic 
obtained from the sensor data is also not provided 
with a systematic name, and even if the sensor data 
is provided with a name, there is a problem that a 
man cannot understand the state indicated by the 
sensor data based on the name, since the name is a 



meaningless mechanical description . 

When a man instructs a robot or a device to do 
something, there is also a problem that an interface 
between the man and the machine accepts only such an 
instruction format as to directly set a value, since 
a machine does not have means for learning the 
correspondence between men and the behavior patterns 
of the machine, or the correspondence in values to be 
changed . 

The present invention is made in order to solve 
the above-mentioned problems, and it is an object of 
the present invention to be able to easily understand 
a state indicated by the sensor data of a machine 
such as a robot, etc. based on a natural language, 
and to operate a machine, such as a robot, etc. , 
using natural words. 

Namely, it is an object of the present invention 
to easily judge a status and to execute a process 
corresponding to the status by providing a new 
classifying means for sensor data, efficiently 
simplifying and storing an enormous amount of data, 
and extracting and determining, the data if judgement 
of the status is needed. It is another object of the 
present invention to provide both language processing 
and device operating means for a machine's learning 
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process and storing a correspondence between an 
instruction from men and the operation of the 
machine, learning to operate according to a command, 
even if the instruction is vague, and storing correct 
5 operations . 

^. Summary of the Invention 

*D A sensor data process method according to the 

m present invention is for processing data obtained 

In 

10 from a sensor, and comprises the steps of: storing 
data groups in a database in which a word is attached 
s to each of the data groups, the data groups being 

L, obtained by classifying inputs from the sensor 

5 directly or after processing; classifying an input 

MB 15 from the sensor using the database to output a 

ffl 

classification result; temporarily storing input data 
from the sensor as data of a new group after 
classifying the input data when it is determined that 
the input data does not belong to any of the data 

20 groups classified in the database; and attaching a 
word to the data of the new group temporarily stored 
to store the data of the new group in the database, 

A sensor data processing apparatus according to 
the present invention is for processing data obtained 

25 from a sensor, and comprises: a characteristic 



extracting unit for extracting a certain 
characteristic from sensor data input from the 
sensor; a characteristic comparing/classifying unit 
for classifying the sensor data based on the certain 
characteristic extracted from the sensor data; and a 
word attaching unit for attaching a word for each 
group of data groups into which sensor data are 
classified when there is a group that a word is not 
attached to in the data groups. 

The sensor data processing apparatus may further 
comprise: a status judging unit for judging a status 
using a certain word attached to a group in the data 
groups, when input sensor data is classified by the 
characteristic comparing/classifying unit as data in 
a group to which the certain word is attached. 

The word attaching unit may request a user to 
designate an appropriate word for a data group of 
sensor data to which no word is attached, and 
attache the appropriate word input by a user. 

A device operation method according to the 
present invention is for processing data with a word 
used in a device for operating an apparatus by 
processing language information, and comprises the 
steps of : storing data indicating a behavior pattern 
of the apparatus in a database in which a word is 
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attached to the behavior pattern; inputting a word 
for instructing an operation of the apparatus; 
comparing the input word and the word attached to the 
behavior pattern in the database; controlling the 
apparatus based on the behavior pattern to which the 
word is attached if the input word matches the word 
attached to the behavior pattern; and controlling the 
apparatus according to a behavior pattern selected 
from a plurality of behavior patterns available for 
the apparatus according to a predetermined rule or at 
random if the input word does not match the word 
attached to the behavior pattern, and when an 
appropriate behavior pattern is executed, attaching 
the input word to the appropriate behavior pattern 
and storing the appropriate behavior pattern in the 
database with the input word, 

A device according to the present invention is 
for operating an apparatus by processing language 
information, and comprises: a database for storing 
behavior patterns of the apparatus to which words are 
attached; an input processing unit for inputting a 
word indicating an operation of the apparatus; a 
comparing unit for comparing the word input by the 
input processing unit and a word attached to a 
behavior pattern stored in the database; a behavior 



selecting unit for selecting a behavior pattern from 
a plurality of behavior patterns available for the 
apparatus according to a predetermined rule or at 
random if the input word does not match the word 
attached to the selected behavior pattern; a behavior 
executing unit for controlling the apparatus 
according to one of the behavior patterns to which a 
matching is detected by the comparing unit and the 
behavior pattern selected by the behavior selecting 
unit; a judging unit for judging whether or not the 
behavior pattern executed by the behavior executing 
unit is appropriate; and a word attaching unit for 
attaching the word input by the input processing unit 
to the behavior pattern judged to be appropriate by 
the judging unit, and storing the behavior pattern in 
the database. 

The input processing unit may convert an input 
word into a word of internal data by way of language 
processing. The behavior selecting unit selects a 
behavior pattern at random when a word is input; the 
behavior executing unit executes a predetermined 
number of or all possible behavior patterns; and the 
word attaching unit attaches the word to a behavior 
pattern which obtains the highest evaluation of a 
certain criterion based on a result of an execution 



10 

of a behavior pattern. 

The device may further comprise: a criteria 
database storing criteria for use in judging by the 
judging unit; an inquiring unit for inquiring which 
criterion is to be used when a behavior is selected 
by the behavior selecting unit; and a judgement 
inputting unit for inputting information on a 
suitability of a behavior pattern executed by the 
behavior executing unit. In this case, the judging 
unit may judge a behavior based on one of the 
criterion obtained by the inquiring unit and the 
information input from the judgement inputting unit. 

The device may further comprise a plurality of 
sensors measuring an operation status of a behavior 
pattern executed by the behavior executing unit, and 
the word attaching unit may store sensor data groups 
obtained from the sensor, the word, and a criterion 
used in a judgement by the judging unit while 
executing the behavior pattern judged to be 
appropriate by the judging unit. 

The judging unit may use a characteristic of the 
sensor data as a criterion. 

The device may further comprise an inquiring 
unit for inquiring of a user an appropriate behavior 
pattern when an evaluation higher than a 
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predetermined level cannot be obtained by the judging 
unit after the behavior executing unit executes a 
predetermined number of behavior patterns, 

A computer-readable storage medium according to 
5 the present invention is for storing programs for 
implementing a data processing method using data with 
_ a word for processing data obtained from a sensor, 

JJ The data processing method comprises the steps of: 

qi when a sensor input is input from the sensor, 

sea 

f^jj 10 classifying the sensor input using a database 

y]> including data groups in which a word is attached to 

= each of the data groups, and outputting a 

L . classification result, the data groups being obtained 

M by classifying inputs from the sensor directly or 

y3 15 after processing the inputs; temporarily storing 

£8 

input data from the sensor as data of a new group 
after classifying the input data when it is 
determined that the input data does not belong to any 
of the data groups classified in the database; and 

20 attaching a word to the data of the new group 
temporarily stored to store the data of the new group 
in the database. 

A computer-readable storage medium according to 
the present invention is for storing programs for 

25 implementing a device operation method for processing 



data with" a Vord used in a device for operating an 
apparatus by processing language information. The 
device operation method comprises the steps of: 
inputting a word for instructing an operation of the 
apparatus; comparing the input word and a word 
attached to a behavior pattern stored in a database; 
controlling the apparatus based on the behavior 
pattern to which the word is attached if the input 
word matches the word attached to the behavior 
pattern; and controlling the apparatus according to 
a behavior pattern selected from a plurality of 
behavior patterns available for the apparatus 
according to a predetermined rule or at random if the 
input word does not match the word attached to the 
behavior pattern, and when an appropriate behavior 
pattern is executed, attaching the input word to the 
appropriate behavior pattern and storing the 
appropriate behavior pattern in the database with the 
input word . 

Brief Description of the Drawings 

Fig.l shows a configuration of the sensor data 
processing apparatus according to the present 
invention. 

Fig. 2 shows a configuration of a device 
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operation apparatus including the language process 
interface according to the present invention. 

Fig. 3 shows another configuration of a device 
operation apparatus including the language process 
interface according to the present invention. 

Fig. 4 shows a hardware configuration for 
implementing the sensor data processing apparatus 
according to the present invention. 

Fig. 5 shows a hardware configuration for 
implementing a device operation apparatus including 
the language process interface according to the 
present invention. 

Fig. 6 shows a case where a moving robot with 
ultrasonic wave sensors is located in a certain 
environment. 

Fig.7A and Fig.7B show an example of sensor 

data. 

Fig. 8 is a flowchart of a characteristic 
extractor unit. 

Fig. 9 is a flowchart of the characteristic 
comparison/classification unit. 

Fig.lOA and Fig.lOB are flowcharts of the word 
provision unit. 

Fig. 11 shows the user interface of the word 
provision unit. 
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Fig. 12 is a flowchart of the status judgement 

unit . 

Fig. 13 shows a data format of the database. 

Fig.l4A and Fig.l4B show another data format of 
the database. 

Fig. 15 shows an embodiment using a neural 
network . 

Fig. 16 shows an example of the characteristic, 
extraction of sensor data. 

Fig. 17 shows an example of sensor data. 

Fig. 18 shows an example of learning by a neural 
network . 

Fig. 19 shows an example of location data. 
Fig. 20 shows a classification result by a neural 
network . 

Fig. 21 explains an embodiment of a device 
operation apparatus including the language process 
interface according to the present invention. 

Fig. 22 is a flowchart of the comparator unit. 

Fig. 23 is a flowchart of the behavior selection 

unit. 

Fig. 24 is a flowchart of the judgement unit. 

Description of the Preferred Embodiment 

In order to solve the above-mentioned problems, 
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•the present invention has, for example, such a 
configuration as shown in Fig.l to Fig. 3. 

Fig.l shows a configuration of a sensor data 
processing apparatus according to the present 
5 invention. 

A characteristic extractor unit 2 extracts 
characteristics from the sensor data obtained from an 

Q 

■yp. information input unit 1 such as a sensor, etc.. A 



characteristic comparison/classification unit 3 
J{] 10 compares a characteristic T extracted by the 

Ln characteristic extractor unit 2 with data groups 

registered in a word attached database (database with 
f7 words) 6, in which a word is attached to each of the 

B data groups. If there is a similar data group, the 

iQ 15 characteristic comparison/classification unit 3 

" reports the corresponding word to a status judgement 

unit 4, while if not, the characteristic 
comparison/classification unit 3 classifies the 
characteristic T into one of data groups classified 
20 in a database (database with no word) 7, (or) creates 
a new category (group) in the database 7 and stores 
the characteristic T in the new category. 

The status judgement unit 4 judges the status 
indicated by the sensor data based on the word of a 
25 data group corresponding to the characteristic T. A 



data group 8 including a word B is the data group 
judged to have the same characteristic T. 

A word provision unit 5 attaches a word that men 
can understand (a human-comprehensible word) to a 
data group of the database with no word 7, and 
registers the data group in the database with a word 
6. A data group deletion unit 10 deletes from the 
database with no word 7 a data group which is not 
provided with a word after the lapse of a certain 
time and is judged to be unnecessary. 

The database with no word 7 is a database for 
storing a characteristic T which does not match with 
a characteristics of the data groups with words 
registered in the database with a word 6, until a 
word is attached or until a predetermined time 
elapses* A database for storing raw data 9 is a 
database for storing data input from the information 
input unit 1 as they are. The apparatus can also be 
configured so that the characteristic data (value) 
extracted by the characteristic extractor unit 2 are 
stored in the database for storing raw data 9 with 
the input raw data. 

One or plural kinds of the information input 
unit 1 can be used, and data input from the 
information input unit 1 which is a target for 



characteristic data extraction can be data at a 
certain time point or a string of data in a time 
series . 

The word provision unit 5 is activated by a 
user's instruction or automatically by a 
predetermined trigger, requests a user to designate 
an appropriate word in order to attach a word to a 
data group of sensor data with no word stored in the 
database with no word 7, and attaches a word input by 
the user to the data group. The word provision unit 
5 attaches a code, which is to be a provisional name, 
to a data group stored in the database with no word 
7 until a word is provided, and replaces the code 
previously attached with a word input by the use. 

When requesting the user to designate an 
appropriate word, if the data group is related to 
static characteristic data, the word provision unit 
5 displays information suggesting to the user that a 
noun should be attached, while if the data group is 
data in a time series and is related to a dynamic 
characteristic data or characteristic data indicating 
a status change, the word provision unit 5 displays 
information suggesting to the user that a verb should 
be attached. Furthermore, for characteristic data 
indicating some status of the sensor, the word 
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provision unit 5 can recommend to the user to attach, 
for example, an adjective. 

A program for implementing the above-mentioned 
process units by a computer can be stored in an 
appropriate computer-readable storage medium such as 
a portable storage medium, a semiconductor memory, a 
hard disk, etc. . 

The characteristic extractor unit 2 can be 
configured by a general -purpose computer and a 
software program, or a part or all of the unit 2 can 
also be configured using a multilayer hierarchical 
neural network in which the input and output have the 
same value. In this case, sensor data are input to 
the hierarchical neural network in order to extract 
characteristic data, and are classified by the input 
and output patterns from an intermediate layer other 
than an output layer. 

The operations of the apparatus shown in Fig.l 
are as follows. 

The characteristic extractor unit 2 extracts a 
variety of characteristics (for example, a shape, a 
color, etc. ) indicated by sensor data input from the 
information input unit 1 such as a sensor, etc.. The 
extracted characteristic T is compared with data 
groups registered in the database with a word 6 by 
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the characteristic comparison/classification unit 3. 
Data groups of the database with a word 6 are 
provided with an appropriate word (word A), 

For example, if it is judged according to 
5 certain criteria that the characteristic T matches 
with one of the data groups registered in the 
database with a word 6, the flow is branched into 

Q 

03 "Matched", and the state or condition of the sensor 

is judged by the status judgement unit 4. In Fig.l, 
j£j 10 it is assumed that the characteristic T matches with 

U1 a data group 8 indicated by a word B. 

Q 

If it is judged by the characteristic comparison 

5,: 

f7 /classification unit 3 that the characteristic T does 

Q not match with any of data groups in the database 

J3 15 with a word 6, the characteristic T is registered in 

33 

the database with no word 7. In this case, if there 
is in the database with no word 7 a data group whose 
characteristic matches with the characteristic T, the 
characteristic T is added to this data group, while 
20 if not, the characteristic T is registered as a new 
data group. 

An appropriate word is attached to a data group 
in the database with no word 7 by the word provision 
unit 5 according to certain criteria. The word 
25 provision unit 5 attaches an appropriate word 
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representing the characteristic to the data group 
with no word by requesting a user (operator) input. 
The data group to which a word is attached is 
registered in the database with a word 6 . 

A word is attached to a data group corresponding 
to the characteristic of the data group. If the data 
group is an aggregate of data indicating a static 
characteristic (no status change due to time, etc.), 
the user is requested to designate a noun or the 
like. If the data group is an aggregate of data 
indicating a dynamic characteristic accompanied by a 
status change (location change of a moving object, 
etc.), the user is requested to designate a verb or 
the like. If the data group is an aggregate of data 
indicating a characteristic of the state of an object 
(color, weight, etc.), the user is requested to 
designate an adjective or the like. 

In the present invention, by classifying sensor 
data obtained from the information input unit 1 using 
a word or an item which is meaningful for a man 
through the above-mentioned processes, the status 
judgement from the sensor data can be facilitated. 
Specifically, a programming for determining an 
instruction to be executed can be facilitated by this 
word, and the announcement of a status can be 



performed by spoken words, etc., thereby easily 
determining a performance based on the sensor data. 

Figs. 2 and 3 show configurations of device 
operation apparatuses each including a language 
process interface according to the present invention • 
In Figs. 2 and 3, units with the same function have 
the same reference numbers. Fig. 3 shows a 
configuration in which a sensor 50, a criteria 
database 51, and a criteria creation unit 54 are 
added to the configuration shown in Fig. 2. 

An input/output unit 40 includes an input/output 
device such as a microphone, a keyboard, a speaker, 
a display, etc. . 

When a word W such as a word spoken by a man, 
etc., or a code input from a keyboard, is input from 
the input/output unit 40, an input process unit 41 
converts the word W -into a computer-readable code 
string, which is word data Wd, further stores both 
the original word W and the word data Wd in a 
language database 42, and activates a comparator unit 
43. 

The comparator unit 43 compares the word data Wd 
with the word data Wd' of behavior data with a word 
stored in a behavior database with a word 49. If the 
word data Wd and Wd 1 match with each other, the 
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comparator unit 43 activates a behavior execution 
unit 46, and executes the behavior pattern of the 
behavior data with a word, while if not, it activates 
a behavior selector unit 44. 

The behavior selector unit 44 selects one 
behavior pattern A from a behavior database 45 for 
storing a plurality of possible behavior patterns A 
according to a criterion or a standard, and activates 
the behavior execution unit 46 to execute the 
selected behavior pattern. 

The selected behavior pattern is executed by the 
behavior execution unit 46, and the result of the 
execution is judged by a judgement unit 47. As a 
result, if the executed behavior pattern is judged 
not to meet certain criteria, the behavior selector 
unit 44 selects a behavior pattern B different from 
the behavior pattern A previously executed according 
to certain criteria, and activates the behavior 
execution unit 46 to execute the selected behavior 
pattern B . If there is no behavior pattern to be 
selected, the process is terminated. When a judgement 
result meeting certain criteria is not obtained even 
after a predetermined number of behavior patterns are 
executed, the execution of the behavior is 
interrupted or an inquiry unit 52 for issuing 
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inquiries to • the user is activated. Alternatively, 
the apparatus can also be configured so that all 
possible behavior patterns can be executed, and the 
seemingly best behavior pattern Am can be selected by 
the user. 

The behavior execution unit 46 executes the 
behavior pattern A selected by the behavior selector 
unit 44, and activates the judgement unit 47. When 
the behavior pattern A is provided by the comparator 
unit 43, the behavior execution unit 46 executes the 
behavior pattern A, and the execution is terminated. 

The judgement unit 47 verifies both word data Wd 
and the behavior pattern A according to user 1 s 
commands from a judgement input unit 53. If it is 
judged to be OK (verified), the judgement unit 47 
activates the word provision unit 48, while, if it is 
judged to be NG (not verified), the judgement unit 47 
activates the behavior selector unit 44. 

The word provision unit 48 attaches word data Wd 
to the behavior pattern A which meets certain 
criteria in the judgement unit 47, and stores both 
the behavior pattern A and the word data Wd in the 
behavior database with a word 49 . 

The behavior database with a word 49 is a 
database for storing a set of the behavior pattern A 



• 
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and word data Wd attached to the behavior pattern A 
in a certain format, or a database for making a set 
of the behavior pattern A, sensor data obtained from 
a sensor 50 during the execution of the behavior 
pattern A and word data Wd attached to the data, and 
storing the set in a certain format. 

The sensor shown in Fig ,3 is a sensor for 
measuring the execution status of the behavior 
pattern A while the behavior execution unit 46 is 
executing the certain behavior pattern A. 

When, as an execution result of several behavior 
patterns, an appropriate behavior cannot be selected 
for the given word W, the inquiry unit 52 inquires of 
the user what is an appropriate behavior. Inquiries 
are made by spoken words or by means of a menu using 
an input/output unit 40 such as a speaker, a display, 
etc . . 

When there is no criteria for judging whether or 
not the selected behavior pattern is appropriate for 
the given word W in the criteria database 51, the 
user directly inputs a judgement (OK or NG) using the 
judgement input unit 53. 

The criteria creation unit 54 stores criteria 
input by the user in the criteria database 51. 
Alternatively, when a judgement is made directly by 
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the user, in order to create an appropriate 
criterion, the criteria creation unit 54 extracts 
some characteristic from the sensor data, and stores 
the characteristic as a criterion. 

A program for implementing the above-mentioned 
process units by a computer can be stored in an 
appropriate computer-readable storage medium such as 
a portable storage medium, a semiconductor memory, a 
hard disk, etc. 

The operations of the apparatus shown in Figs. 
2 and 3 are as follows . 

A word W (command) given by the user, etc. is 
processed by the input process unit 41, and is 
converted into word data Wd (instruction data) 
comprehensible to a computer. The word W (command) is 
a verbal command, for example, "Speed up!" or "Turn 
right!", and is not needed to be indicated by an 
actual value. The word W and the word data Wd are 
stored in the language database 42 in a format where 
the correspondence between the word W and the word 
data Wd can be recognized. For example, if the word 
W is an audio signal, the input process unit 41 
stores audio signals input from a voice inpuit device 
such as a microphone, etc., or extracts the 
characteristic of the audio signals, and outputs the 
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extracted data as word data Wd. 

Although usually the word W is a natural word, 
word data Wd are simply an internal code for a 
computer corresponding to the word W and do not have 
5 a meaning as a natural word. A computer does not have 
to know the meaning of the word W. By providing the 
input process unit 41 with a speech recognition 
function, a voice input can also be processed in the 

gg same way as a character input from a keyboard, etc. 

m 

10 Alternatively, for example, if a dictionary of 
synonyms, etc. is provided and the input word W is 

Q 

s converted into another word W 1 with the same meaning, 

^ word data corresponding to the word W 1 can also be 

y used for the processes after that. 

15 After being processed by the input process unit 

41, the word data Wd are compared with data stored in 
the behavior database with a word 49 by the 
comparator unit 43. If there are no data in the 
behavior database with a word 49, no comparison is 
20 made, and immediately the behavior selector unit 44 
is activated. 

If there are behavior data with a word in the 
behavior database with a word 49 and the word data Wd 
match with one of the behavior data with a word, the 
25 behavior pattern is executed by the behavior 
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execution unit 46, and after the completion of the 
execution the instruction is terminated. 

On the other hand, if the word data Wd do not 
match with any of the behavior data with a word, the 
behavior selector unit 44 is activated. The behavior 
selector unit 44 selects an appropriate behavior 
pattern out of possible behavior patterns for the 
apparatus stored in advance in the behavior database 
45 (for example, all instructions for implementing 
conventional instructions such as "Speed up by a 
certain degree!", "Change the direction by a certain 
degree!", "Change the temperature by a certain 
degree!", etc.), and requests the behavior execution 
unit 46 to execute the behavior pattern. The behavior 
execution unit 46 executes the requested behavior 
pattern. 

As a rule, the behavior result is judged by the 
user. For the judgement, if a behavior selected by 
the apparatus corresponds to an instruction given by 
a user, OK is input from the judgement input unit 53, 
while, if not, NG is input. The input from the 
judgement input unit 53 is a simple input such as 
"The execution result is right or wrong.", and is 
input from the input/output unit 40 such as a 
keyboard, a switch, etc. The appropriateness of the 
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behavior can also be determined . by a user inputting 
a word such as "right" or "wrong" from the input 
process unit 41 a certain time after the completion 
of the behavior. In this case, behavior data will be 
"Judge that the previous behavior is right!", etc. 

Since in this judgement method a user always has 
to monitor the behavior state of the apparatus, the 
method is complicated. If a user can create criteria 
in advance and when the behavior meets the criteria, 
the behavior can be judged to be OK, and the method 
is more convenient. Thus, a user makes an inquiry 
about criteria using the inquiry unit 52. The 
obtained criteria are stored in the criteria database 
51. When a state where given criteria are met is 
realized, the judgement unit 47 judges that it is OK. 
In this case, the judgement is made by comparing 
sensor data. Thus, an apparatus is configured so that 
criteria can also be obtained from sensor data. For 
example, if an instruction "Increase the speed (V)! M 
is issued, an acceleration a (> 0) detected from the 
output of the sensor 50 becomes the criterion. 

The criterion can be laid down on what should 
happen in the behavior pattern stored in the behavior 
database 45. For example, if a behavior pattern is 
about changing the speed V, it is sufficient that 



29 

acceleration a becomes positive or negative. 
Accordingly, the criterion is either "acceleration a 
> 0" or "acceleration a > 0" . 

When a user, etc, directly judges without using 
criteria, the criteria will be vague. Thus, an 
appropriate characteristic data (value), for example, 
an average speed, an acceleration, etc. is calculated 
from sensor data obtained when an OK'd behavior is 
executed, and are stored in the criteria database 51 
as a criterion. 

If as a result of the judgement it is judged to 
be OK, the word provision unit 48 is activated, an 
executed behavior and the word data Wd of a given 
verbal instruction are combined, and the combination 
of the behavior and the word data Wd are stored in 
the behavior database with a word 49 as behavior data 
with a word. That is, data such as "Instruction: 
Speed up! -» Behavior: Acceleration a > 0" are stored. 

On the other hand, if it is judged to be NG, the 
behavior selector unit 44 is activated to select a 
next behavior. In this case, the behavior selector 
unit 44 selects and executes a behavior other than 
those executed before. 

Furthermore, the criteria according to which 
behavior data with a word is judged to be OK are 
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stored in the criteria database 51 in such a way that 
the criteria can correspond to the word. Thus, which 
criterion should be used when a certain instruction 
is issued is learned one by one. Accordingly, the 
5 criteria can be automatically used without inquiring 
of a user. 

Fig. 4 shows a hardware configuration for 
implementing the data sensor processing apparatus of 
the present invention. 

10 For the information input unit 1 shown in Fig.l, 

for example, a sensor 111 is used, and a sensor 
signal is obtained by a computer 112. The obtained 
data and characteristic data are stored in a storage 
unit 113, and the database with a word 6 and the 

15 database with no word 7 are realized as the storage 
unit 113 such as a hard disk, etc. The processes such 
as characteristic extraction, characteristic 
classification, status judgement, etc. are executed 
by the computer 112. The results of the status 

20 judgement are presented to a user using a display 
unit 114, a speech synthesizer unit 115, etc. When 
an actuator 116 is controlled by the result of the 
status judgement according to control information 
selected by the result of the status judgement, for 

25 example, some work such as a forward movement, 



backward movement, rotation, etc. is executed. 

The display unit 114 and the speech synthesizer 
unit 115 are also used for the inquiry to a user at 
the time of providing a word. When a user inputs a 
word, the word is input using a keyboard, a mouse or 
a speech recognition unit, which are all kinds of 
sensors. Therefore, all input devices to these 
computers 112 are shown as sensors 111. 

Each computer 112 comprises one or a plurality 
of processors (CPU) and a memory for storing programs 
and data for control. The computer 112 can also 
comprise a neuro-computer such as a hierarchical 
neural network, etc. 

Fig. 5 shows a hardware configuration for 
implementing a device operation apparatus with the 
language process interface of the present invention. 

An input unit 121 is an input, device for 
inputting an instruction from a user, such as a 
microphone, a keyboard, etc. or an input device for 
inputting the judgement by a switch, etc. The input 
data are transmitted to a computer 123. A sensor 122 
is a sensor for measuring the status during the 
execution of a behavior, and the data are received by 
the computer 123. 

The computer 123 executes the program according 
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to the present invention, processes an input audio 
signal and a sensor signal, and stores the result of 
the process in a storage unit 124. 

The storage unit 124 stores a variety of 
databases and the program of the present invention, 
and comprises a hard disk, a semiconductor memory, 
etc. A speech synthesizer unit 125 issues a spoken 
word ( or voice ) when there is an instruction which 
requires a spoken word, and synthesizes and issues 
the spoken word when inquiring of the user. 

The display unit 126 is used in the same way as 
described above, and displays a processing state 
during the execution of a variety of processes. A 
device 127 is a device such as a robot, an actuator, 
air conditioner, etc., and is operated by an 
instruction from the user. The input unit 121 through 
the display unit 126 can be made separately from the 
device 127, and it can also be made as a hardware 
configuration wherein the input unit 121 through the 
display unit 126 are incorporated into the device 
127. 

A mobile (travelling) robot for which the 
present invention is applied is described below with 
reference to the drawings. 
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1. Sensor data processing apparatus 

First, the sensor data processing apparatus of 
the present invention is described. 

Fig. 6 shows that a mobile robot 30 with an 
ultrasonic wave sensor is located in a certain 
environment. Although only ultrasonic sensor data are 
considered here, the sensor data of the present 
invention are not necessarily limited to this kind. 
For example, sensor data including both the location 
(X, Y) and direction 9 of a robot 30 can also be 
used. This mobile robot 30 includes an ultrasonic 
wave sensor (or ultrasonic sensor) 31 for the 
information input unit 1 in Fig.l. Eight ultrasonic 
wave sensors 31, for example, are provided around the 
body of the robot 30, and the distance between the 
sensor and a certain object can be measured. 

In the environment where a robot is located in 
this example includes a corridor 33 surrounded by 
walls (or dividers) 32, a right corner 34, a T-shaped 
junction 35, a left corner 36 and a blind alleys 37, 
etc. at the end of the corridor 33. 

The value of sensor data measured at a certain 
location using the eight ultrasonic wave sensors 31 
is assumed to be U. U is a vector consisting of eight 
sets of distance data. The robot 30 moves, and the 



34 

values of the ultrasonic wave sensors 31 are measured 
at intervals of a sampling time. 

Since the distance data in a place such as the 
corridor 33 obtained by the ultrasonic wave sensors 
31 are short to the left and right, and long to front 
and back of the robot 30, the outputs of the 
ultrasonic wave sensors 31 are expected to be in such 
a state as mentioned above. While moving through the 
corridor, the robot 30 obtains a lot of ultrasonic 
wave sensor data of similar values. 

Fig.7A and Fig. 7B show an example of the 
ultrasonic wave sensors data. 

It is assumed that there are eight ultrasonic 
wave sensors for measuring the distances of 8 objects 
located in the directions of 1 through 8 from the 
robot, and all the sensors can measure a distance of 
up to 10m. For example, in a rectangular space 2m 
wide and 20m long as shown in Fig.7A, the ultrasonic 
wave sensor data corresponding to locations A, B and 
C in the diagram become as shown in Fig.7B. In this 
case, it is assumed that the radius of the robot is 
0,4m, and the ultrasonic wave sensors are set on the 
surface of the robot. 

Next, a concrete example of each process unit 
composing the present invention is described in 
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detail, 

( 1 ) Information input unit 

When data is input from an information input 
5 unit 1 (ultrasonic wave sensor 31 in this 
embodiment), the data is transmitted to a 
characteristic extractor unit 2. If a characteristic 

O 

yp extraction method is predetermined, a characteristic 

rg extraction is performed according to this method. Raw 

10 data from which a characteristic was extracted are 

y = 

discarded or stored in a database for storing raw 
s data 9, Since there is also a case where a 

FT characteristic is extracted from time series data, it 

Q is desirable to store the raw data, and basically the 

yg 15 input data are stored in the database for storing raw 

^ data 9. 

( 2 ) Characteristic extractor unit 

The characteristic extractor unit 2 is provided 
20 with a predetermined characteristic extraction 
method. Fig. 8 is a flowchart showing the operation of 
the characteristic extractor unit 2. Two or more 
characteristic extraction methods may be used. In 
this case, an extracted characteristic data becomes 
25 vector data. Here, it is assumed that n sets of 



36 

characteristic data T 0 through T n . x are extracted; 
First, sensor data are input (step SI in Fig.8), and 
i is initialized to 0 (step S2). Then, characteristic 
data T L is extracted (step S4), and the extraction of 
the characteristic is repeated while incrementally 
increasing i (step S5), and the extraction is 
terminated when i reaches n ( step S3 ) . 

At this time, the input data themselves can also 
be taken as the result of the characteristic 
extraction process without extracting any 
characteristics. In this case, the sensor data 
themselves are regarded as a characteristic. However, 
since the capacity of data to be stored becomes 
large, this method may not be efficient. Therefore, 
here, a characteristic of the sensor data such as 
that the distance is long to the front and back, and 
short to the left and right, is assumed to be 
extracted. For the characteristic data, a ratio of 
the front-to-back distance to the left-to-right 
distance of the ultrasonic wave sensor data is used. 
This characteristic data T, for example, is 
calculated using the following equation. 



T = (Ul + U5) / (U3 + U7) 



• * 
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Un is the n-th data of the ultrasonic wave 
sensor. If the data in Fig.7B are substituted into 
this equation, characteristic data (value) T A at 
location A, characteristic data T B at location B, and 
characteristic data T c at location C become 12.2, 
16.0, and 12.2, respectively. When the robot is 
located at a crossroads or at the center of a wide 
space, the characteristic data T become a value near 
to 1, because the front-to-back distance and the 
left-to-right distance are almost the same. 
Therefore, if original ultrasonic wave sensor data 
are classified according to a criterion of, for 
example, T > 10, a value of characteristic data T 
which is a standard value for judging whether or not 
the robot is located in a corridor can be obtained. 
Then, the obtained value of characteristic data T is 
transmitted to a characteristic comparison 
/classification unit 3. 

Then, when new sensor data are input, a 
characteristic extraction is performed in the same 
way as described above, and the extracted 
characteristic data T is transmitted to the 
characteristic comparison/classification unit 3. 
Further, raw data are stored in the database for 
storing raw data 9. The characteristic data can also 
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be stored in -the database for storing raw data 9 
together with the raw data. 

The characteristic data extracted by the 
characteristic extractor unit 2 can be any 
characteristic, if the characteristic is obtained 
from target data, and if there is no appropriate 
characteristic data, the input data themselves can be 
used as characteristic data. 

There are two kinds of sensor data; static data 
with no relation to time and time series data 
changeable with a lapse of time. 

The static data with no relation to time provide 
a single characteristic by themselves. Generally 
speaking, the ratio of RGB over a whole screen, an 
area expressed in a binary system, a number of edges 
in a picture, etc. can be used as characteristic data 
of picture data. When ultrasonic wave sensors are 
provided in four directions of the front, back, left, 
and right, the ratio between the distances of the 
four directions, for example, a distance ratio of the 
front-to-back direction to the left-to-right 
direction, such as "a front-to-back distance is 
longer than a left-to-right distance", "only a front 
distance is longer than the back, left, and right 
distances", etc. is used for characteristic data. 



These are examples of the characteristic extraction 
method from static data. 

In the case of time series data, the change with 
the passage of time of obtained data is extracted as 
characteristic data. For this, the time differential 
of sensor data obtained from hour to hour is used. 
For example, a differential of ratios of RGBs over 
the whole screens between two pictures at the times 
of t and t+At, a differential of the areas expressed 
in a binary system between the two pictures, a 
differential of the numbers of edges in the two 
pictures, etc. are used. When ultrasonic wave sensors 
are used, a differential between two sensor data 
obtained at the times of t and t + At are used, if 
attention is paid to the status change. The speed or 
acceleration of the movement of a robot, which 
already includes a time element, can also be used as 
characteristic data. 

(3) Characteristic comparison/classification unit 

The characteristic comparison/classification 
unit 3 comprises a characteristic comparator unit for 
comparing the input data with word attached data 
groups registered in a database with a word 6, and a 
characteristic classification unit for registering 
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-the input data in a database with no word 7, if there 
are no same data groups among the word attached data 
groups, 

5 (3.1) Characteristic comparator unit 

The characteristic comparator unit receives an 
extracted characteristic data (value) Ti (i = 0, 
n-1) or when there is no characteristic data 
extraction, the ultrasonic wave sensor data 

10 themselves, and compares the characteristic data Ti 
or the ultrasonic wave sensor data with k data groups 
in the database with a word 6, and judges whether or 
not there is a corresponding data group. Each of the 
k data groups have m g elements. m g is not necessarily 

15 constant. Fig. 9 is a flowchart showing this process. 

In a characteristic comparison, first, the 
characteristic data (value) Ti is input (step Sll in 
Fig.9). Then, a loop variable g for counting the 
number k of the data groups in the database with a 

20 word 6 is initialized to 0 (step S12). Then, the 
similarity between characteristic data Ti whose i is 
0 to (n-1) and characteristic data Tj whose j is 0 to 
(m 9 - 1) are compared (steps S14 through S20 ) while 
incrementing g by one (step S21) until the variable 

25 g exceeds k (step S13). Here, Tj is the j-th 
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characteristic data of the g-th data group in the 
database with a word 6 . 

Since at the time of initializing, there are no 
data in the database with a word 6 (k = 0), there is 
no target for comparison, and thereby the result of 
the comparison becomes "not matched" Then, the 
characteristic data (value) T is transmitted to the 
characteristic classification unit as it is. If there 
are some data in the database with a word 6, the 
similarity between two characteristic data is 
calculated, and it is judged whether or not the two 
characteristic data match with each other. The 
similarity of the two characteristic data T, for 
example, is calculated using the distance L between 
the two values of the characteristic data T according 
to the following equation. 

L = (T 4 - Tj) 2 (i * j) 

If characteristic data is expressed as a vector, the 
distance L is the sum of the distances of each 
element. If the distance L is less than a certain 
threshold e, those sensor data are judged to be 
"matched". Characteristic data (value) Ti judged to 
be "matched" is stored in a corresponding data group 
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in the database with a word 6, and simultaneously is 
transmitted to a status judgement unit 4. 
Characteristic data judged to be "not matched" is 
transmitted to the characteristic classification 
unit, 

A concrete similarity judging method in the case 
where raw data are used as they are is described 
below taking ultrasonic wave sensor data as an 
example. The ultrasonic sensor data are assumed to be 
a vector consisting of eight elements. First; input 
ultrasonic wave sensor data are assumed to be Dl. In 
order to search data similar to Dl, one set of target 
data for search is selected, which is assumed to be 
Dt. The similarity degree between Dl and Dt, for 
example, is calculated by the following distance 
calculation equation. 

L = (Dl - Dt) 2 

It can be estimated that the smaller the value 
of the distance L is, the more similar Dl and Dt are. 
Sensor data other than ultrasonic wave sensor data 
can also be processed in the same way. For data from 
other sensors, similarity can be judged in the same 
way, if the arrangement of each element is set to be 
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■the same. 

When the input data are compared with data 
groups classified according to a criterion of 
"characteristic data T > a (a is a constant)", it is 
sufficient if it is judged whether or not T > a. 

(3.2) Characteristic classification unit 

The characteristic classification unit of the 
characteristic comparison/classification unit 3 
searches for characteristic data similar to 
characteristic data T in the database with no word 7 . 
Although the process procedure of the characteristic 
classification unit is the same as the flowchart 
shown in Fig. 9, characteristic data Tj with which 
input data are compared is selected from the database 
with no word 7 . In the database with no word 7 there 
are data groups classified by some characteristic 
data. Then, a similarity judgement is performed 
against arbitrary data composing each data group. 
Here, all data can also be compared. For the 
similarity judgement, for example, the distance L 
between the values (two sets of data) is used. 



L - (T ± - Tj) 2 



(i + j) 
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If the distance L is less than a certain 
threshold €, those sensor data are judged to be 
similar. If new data are judged to be "similar" 
against arbitrary data belonging to a certain data 
group, the new data are registered in that data 
group. If the new data are judged to be "not 
similar", the new data are registered in the database 
as a new data group. 

For the classification of raw data from which a 
characteristic is not extracted, a similarity 
judgement is available from the distance calculation 
of each element of a sensor used in the 
characteristic comparator unit. 

In another classification method, classification 
is performed according to a criterion of what range 
of values the characteristic T is in. For example, 
input data can also be classified according to a 
criterion of "characteristic data T > a (a is a 
constant)". This criterion is used, for example, when 
input data are classified as follows; if a speed is 
positive, move forward, if negative, move backward, 
and if zero, stop . 

(4) Word provision unit 

The word provision unit 5 attaches a word to 
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data groups in -the database with no word 7 in which 
the data reach a certain volume. Figs. 10A and 10B 
are flowcharts showing the processes of the word 
provision unit 5. As sensor data are input, the 
composing data number of a data group in the database 
with no word 7 increases gradually. When the data 
number exceeds a certain volume p, the data group is 
provided with an appropriate word. When attaching a 
word, for example, the word provision unit 5 inquires 
of a user what name should be attached to the data 
group by providing sensor data and indicating the 
state of the sensor data (in the case of a robot, the 
location of the robot ) . 

Namely, as shown in Fig.lOA, the word provision 
unit 5 checks whether or not the element number m g of 
data groups of the first through k-th groups in the 
database with no word 7 is greater than a certain 
volume p (steps S31 to S34), and if the element 
number m 9 is greater than the certain volume p, the 
word provision unit 5 activates the inquiry routine 
shown in Fig.lOB (step S35). 

In the inquiry routine the word provision unit 
5 judges whether or not a user is at hand (step S41 ) , 
and if a user is not available (step S41: NO), the 
word provision unit 5 terminates the process. If a 
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user is available (step S41: YES), the word provision 
unit 5 inquires of the user ( step S42 ) , and requests 
the user to input a word for the sensor data ( step 
S43 ) . The word provision unit 5 attaches the input 
5 word to the data group (step S44), and registers the 
data group in the database with a word 6 (step S45 ) ♦ 
In the inquiry in step S42, for example, the contents 
of the data are indicated using such a user interface 

H" as shown in Fig. 11, and a word is input. 

05 

LR 10 The details on this provision of a word are 

m 

further described below. Although, when attaching a 
word, it is desirable to use a human language as much 
M= as possible, it is difficult to automatically attach 

g such a name to data, unless the apparatus is provided 

^ 15 with such knowledge in advance. Thus, a word is 

65 attached to the data group by the apparatus 1 

appropriately inquiring of a user using the user 
interface shown in Fig. 11. 

The apparatus presents the classification 
20 criteria and the data contents of data groups 
classified in advance, and requests a user to attach 
a word. For example, for characteristic data of the 
rotating speed of a robot Ae / At, a classification 
criterion of Ae / At > 0 can be used. Furthermore, 
25 since this is a time series characteristic, the 
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advice "it is appropriate to attach a verb to this 
data group.", etc. is also given via the user 
interface. 

The user attaches the word "turn" to the data 
group according to this classification criterion. In 
this case, although the words "turn left" are more 
appropriate for A 9 / At > 0, the user does not 
usually chose the word "left". Then, when a 
classification criterion of Ae / At < 0 is presented, 
the user tries to attach a word "turn" again. 
However, since the word "turn" is already used for 
another data group, the apparatus can suggest to the 
user that there are two kinds of data groups of both 
Ae / At > 0 and Ae / At < 0 by giving the advice "the 
word turn is already used for another data group with 
a characteristic of Ae / At > 0". At this time, more 
precise words of "turn left" and "turn right" can be 
attached to data groups of A9 / At > 0 and Ae / At < 
0 , respectively . 

Furthermore, since the apparatus can recognize 
that the user has tried to attach the same word to 
these two data groups, the apparatus can estimate 
that there is some relationship between these data 
groups. Thus, the apparatus can inquire of the user 
whether or not a word should be attached in order to 
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further differentiate these data groups* At this 
moment, since at first the user has tried to attach 
the word "turn" to these two different data groups, 
it can also be estimated that this word is common to 
these data groups. Thus, the apparatus can inquire of 
the user whether or not to attach the word "turn" to 
these data groups. If the user replies OK to the 
inquiry, the word "turn!" is attached to these two 
data groups. Otherwise, the apparatus requests the 
user to attach another word to the data. 

After this, the classification criteria for 
distinguishing these two data groups become A9 / At 
> 0 and AG / At < 0. 

If a user is not available, an appropriate code 
is attached, and is modified later by inquiring of a 
user, if needed. If a user is not at hand, an 
appropriate word is attached by the apparatus. Of 
course, no word can also be attached. Whether or not 
a user is at hand can be judged from the fact that 
there is a reply within a certain time after an 
inquiry is issued, or can be verified by using a 
sensor of infrared rays (an infrared sensor), etc.. 

Then, when it is found that a user is now at 
hand, the word provision unit 5 activates an inquiry 
unit for attaching a word to a data group about which 
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an inquiry was not available, and requests the user 
to attach a word to the data group. 

A data group with a word attached is then moved 
from the database with no word 7 to the database with 
a word 6, and stored there. Raw data stored in the 
database for raw data 9 are provided with a code from 
which it can be judged to which data group the raw 
data belong. 

The characteristic data obtained in this way are 
classified and registered in the database with a word 
6 in order. 

Although the word provision unit 5 attaches a 
word using an appropriate user interface, the word 
provision unit 5 also attaches "noun", "verb", 
"adjective", etc. to the data group according to the 
characteristic of the data group. The criteria in 
this case are shown below. 

(1) Provision of a noun 

Since the name of a thing indicates the static 
state of the thing, a noun is attached to data 
classified according to a static state, that is, not 
time-series data. 



(2) Provision of a verb 
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Since a verb indicates a change with the passage 
of time and an operation, a verb is attached to data 
classified according to a time-series characteristic. 

(3) Provision of an adjective } 

An adjective indicates the volume and property 
of a characteristic, that is, for example, "the 
operation is quick.", "the volume is large.", etc. 
Therefore, when there is characteristic data and the 
difference between properties is described, an 
adjective is used. Here, it is assumed that a robot 
moves, and ADir / At indicates the characteristic 
data of a movement translation. If ADir / At obtained 
from one sensor signal series is larger than the ADir 
/ At obtained from another sensor signal series, the 
former is called higher in speed than the latter. 
Conversely, the latter is called lower in speed than 
the former. Alternatively, if a certain threshold is 
provided and ADir / At is greater than this 
threshold, the word "high" can also be attached. If 
ADir / At is smaller than the threshold, the word 
"low" can also be attached. 

Furthermore, a word can also be attached 
according to fuzzy logic. In this case, membership 
functions corresponding to "high", "normal" and "low" 
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are defined; and one of "high", "ordinary" and "low" 
is selected according to the aggregate degree. In 
this case, not only ADir / At but also the 
definitions of the membership function are added as 
characteristic data. 

(5) Status judgement unit 

A status judgement unit 4 refers to both 
characteristic data during judgement and a word 
attached to a corresponding data group in the 
database with a word 6, and judges the current status 
from the contents. For example, if referring to a 
data group with the word "corridor" and judging that 
the current characteristic data correspond to the 
word "corridor", the status judgement unit 4 judges 
the state to be "in the corridor". That is, the 
status judgement unit 4 judges the status of input 
data by comparing the input data with data stored in 
the database with a word and further using the name 
attached to the word data. 

Fig. 12 is a flowchart of the status judgement 

unit. 

The status judgement unit 4 obtains the data 
name (word) of a matched characteristic data list 
from the database with a word 6 (step S51). The 



52 

status judgement: unit 4 indicates the current status 
by the obtained data name (step S52), and displays 
the status (step S53). The status judgement unit 4 
executes processes corresponding to the status, if 
needed (step S54). For example, if the characteristic 
data of data obtained from the ultrasonic wave sensor 
of a robot is a "left corner", a control such as 
"after turning left 90 degrees, move forward.", etc. 
is performed. 

( 6 ) Data group deletion unit 

In the data stored in the database with no word 
7, data which do not reach a predetermined number of 
pieces of data, even after a certain time elapses, 
have a good possibility of including an error due to 
some cause, such as noise, etc. Since such data 
cannot be called data with reliability sufficient to 
attach a word, the data are deleted, which is 
performed by a data group deletion unit 10. 
Alternatively, these data may have some peculiar 
property. In that case, the data do not have to be 
deleted. 

( 7 ) Database for storing raw data 

Fig. 13 shows the data format stored by the 
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database for storing raw data. 9. 

The database for storing raw data 9 is a 
database for storing data obtained from the 
information input unit 1 as they are, and, as shown 
in Fig. 13, it stores the identification number of 
data automatically provided by a system, the 
acquisition time of data, the kind of a sensor and 
the contents of data. This example is configured so 
that both the kind of characteristic data and the 
characteristic data can be stored in the database for 
storing raw data 9, In the case of time-series sensor 
data too, the identification number of data, the 
acquisition time of data, the kind of a sensor and 
the contents of data are stored in the same way. Both 
the kind of characteristic data and the 
characteristic data can also be stored in the same 
way. 

( 8 ) Database with no word/Database with a word 

Fig.l4A shows a data format of the database with 
no word. 

The database with no word 7 stores the 
identification number of classified data groups, the 
number of pieces of data belonging to the group, a 
provisional word attached to the group, the 
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provisional category of the word, the kind of 
characteristic data being a classification criterion, 
the criteria, and the identification number of data 
belonging to this group. The provisional word 
attached to the group and the provisional category of 
the word are automatically attached by the apparatus 
according to the status of the data. 

Fig.l4B shows a data format of the database with 
a word . 

The database with a word 6 registers data groups 
in the database with no word 7 which are provided 
with a word by the word provision unit 5. The data 
format is the same as that of the database with no 
word 7, but the database with a word 6 does not have 
a provisional word or a provisional category, but 
information on a word and a category attached by the 
word provision unit 5. 

( 9 ) Embodiment using a hierarchical type neural 
network 

Next, an example of extracting characteristics 
using a hierarchical neural network is described 
below. Although the characteristic extractor unit 2 
can be implemented by using a general-purpose 
processor, the characteristic extractor unit 2 can 



also be implemented by using a neuro-computer of a 
hierarchical type neural network. 

Fig. 15 shows a method for extracting 
characteristics using five layers of a neural 
network . 

This hierarchical type neural network is made to 
learn by providing teaching data so that an input 
value and an output value can become the same, that 
is, to learn an identity map. The robot 30 including 
8 ultrasonic wave sensors 31 as shown in Fig. 6, is 
made to learn so that an input value and an output 
value can always be the same using 8 values from one 
ultrasonic wave sensor as the input to a neural 
network and 8 of the same ultrasonic wave sensor 
values as teaching data. In this case, the 
characteristic data of the data is the output of an 
intermediate layer of a neural network. 

The output value of each unit of the neural 
network is adjusted to be ranged between ±1. Here, 
the output of the third layer is assumed to be used. 
Since, generally speaking, there are a plurality of 
intermediate layers, the characteristic data becomes 
a vector consisting of these outputs. In the example 
shown in Fig. 15, the output unit of the third layer 
consists of two units, and the output of each of 



these units ( unit 0 and unit 1 ) becomes the 
characteristic data. The similarity between the 
characteristic data is calculated using the above- 
mentioned distance L. 

Fig. 16 shows the state where the mobile robot 
shown in Fig. 6 is located at a variety of positions 
along a corridor (12 positions: (A) through (M)). 

Fig. 17 shows the values of each of the 8 
ultrasonic wave sensors mounted on the robot. 

Fig. 18 shows a plotting map indicating the 12 
sets of data shown in Fig. 17 after learning by the 
neural network, in which the output values of unit 0 
and unit 1 of the intermediate layer of the third 
layer are plotted along the horizontal and vertical 
axes, respectively. It is found from Fig. 18 that 12 
kinds of sensor data are distributed appropriately. 
Therefore, data within the range of a certain 
distance, for example, 0.2 from each point, can be 
classified. 

Fig. 19 shows the output values of each unit at 
that time. 

Fig. 20 shows a plotting map indicating the 
output data of the third layer in the same way as 
shown in Fig. 18. The output data is obtained from a 
pre-learnt neural network according to the input data 
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of the pre-learrvt neural network, which are the 
ultrasonic wave sensor data obtained when the robot 
moves in the corridor shown in Fig. 6, Outputs 
corresponding to the learnt data and newly classified 
data are indicated by a black point (§) and a diamond 
mark (0), respectively. As a result, target data for 
classification are found within a distance of 0.2 
except for a part of data. Therefore, a status 
judgement can be performed using a name attached to 
the learnt data, for example, "crossroads" for (A), 
"corridor" for (B), etc. 

In this embodiment, since data to be stored in 
the database with a word 6 (corresponding to the 
original 12 sets of data) are designated in advance 
by a user and the words are also attached, both an 
efficient word setting and status judgement are 
available. 

(10) When time-series data are used as characteristic 
data 

Next, an embodiment in the case where time- 
series data are used as characteristic data is 
described below with reference to an example where a 
mobile robot moves. A mobile robot usually has two 
kinds of motors for both a forward/backward movement 
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and a rotation (or steering) . These two motors are 
provided with sensors for detecting their rotation, 
for example, an encoder. Under these circumstances, 
the robot is moved by orders, and the locations of 
the robot during movement are measured in sequence. 

Since, when the motor for forward/backward 
movement drives, a value of the encoder changes, the 
status change is detected. That is, the status change 
of the encoder is extracted as characteristic data by 
the characteristic extractor unit 2. This 
characteristic data can be set according to the 
forward/backward direction of the movement, that is, 
the codes of " + " for forward and "-" for backward. 
The rate of change, that is, a speed, can also be 
detected as characteristic data. That is, the 
characteristic data obtained are data of the speed 
and the code . 

The speed and code data are transmitted to the 
characteristic comparison/classification unit 3. The 
speed and code data are compared with data groups 
relating to speed and direction with reference to the 
database with a word 6. Then, a word is attached so 
that the word of "forward" or "backward" can be 
selected according to the code of " + " or 
respectively. Then, the status judgement of a 
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forward/backward movement is available. 

The detailed description is as follows. Since 
the location data of the robot changes as the robot 
moves, the location data are obtained in a time- 
series. If the obtained data groups are assumed to be 
SI, ... , Sn, these are time-series data obtained at 
certain intervals of At. Here, a differential AS 
between each item of data is studied. 

AS ± = (S ± - S^) / At i = 1,..., n 

where AS ± indicates how much the location of the robot 
changes in At. The data are classified according to 
the classification method described earlier using 
this as characteristic data. The posture of the robot 
is also taken into consideration, and the elements of 

5 are assumed to be x, y and 0. x and y indicate 
movement amount in a two-dimensional plane of the 
robot, and 9 indicates the rotation amount of the 
robot. AS indicates the state of rotation when only 

6 changes, and indicates a position translation when 
only x and y change. That is, if similar data are 
selected according to a criterion of AG > 0, this can 
be judged as "during turning left", and for example, 
the data aggregate is named "turn left". If Ae < 0, 
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■the data aggregate is named "turn right". 

Next, the case where A0 = 0 is studied. In this 
case, the moving direction of the robot is indicated 
by Dir = AT AN (y/x), and it can be judged that, if 
the moving direction Dir indicates the direction in 
the front of the robot, the robot is in a state of 
"moving forward", while if Dir indicates the 
direction in the rear of the robot, the robot is in 
a state of "moving backward". If A8 = 0, Ax = 0 and 
Ay = 0, the robot is halted, and if both Ax and Ay 
have certain values, the robot rotates while 
translating, that is, turns in a curve. 

Similarities between data are calculated in the 
above-mentioned way, and if a similarity is found 
between two set of data, a certain kind of flag is 
attached to the data. This flag is attached so that 
those data can be judged to belong to the same group 
when being detected. If similarity is not found, 
another flag different from that of the 
characteristic data first obtained is attached. 

In this way, every time sensor data are input, 
characteristic data is extracted, classified and 
stored . 

Although embodiments where the present invention 
applies to a mobile robot are mainly described so 



far, "the present invention is not limited to a mobile 
robot, but can also be applied to a system for 
executing some process or judgement using an input 
from a sensor. 

2 . Device operation apparatus with a language process 
interface of the present invention 

Next, a device operation apparatus with the 
language process interface of the present invention 
is described with reference to Fig. 21. 

Here, the description is made taking an example 
of the process in the case where a user gives an 
instruction "speed up!" to a mobile robot. 

When words are issued by a user, the voice data 
are received by an input/output unit 40 such as a 
microphone, etc. , and transmitted to a computer. In 
the computer an input process unit 41 processes the 
data. In the case of voice data the input process 
unit 41 stores the characteristics of the voice, for 
example, a power spectrum for each band, a 
voiceprint, etc. in a language database 42. The name 
of the stored data is assumed to be Wd. Wd is 
compared with behavior data stored in the database 
with a word 49 . 

In the case of a voiceprint the similarity of 
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the voiceprint is compared using a distance between 
voiceprints. As a result of the comparison, if there 
is no matching or if there are no data in the 
behavior database with a word 49, a behavior selector 
unit 44 is activated. If there is a matching, a word 
is selected from the behavior database with a word 
49, and combined behaviors are executed. 

For example, if a behavior matching with an 
instruction "speed up!" is acceleration a = 1 
(meters/second 2 ), this behavior is selected and 
executed, and the speed gradually increases. As a 
result of the comparison, if there is no matching, 
the behavior selector unit 44 is activated, and 
appropriate behavior data are selected out of 
behavior data stored in the behavior database 45, 
such as acceleration a = -1 ( meter/ second 2 ) , 
acceleration a - 0 (meter/second 2 ), acceleration a = 
1 ( meter / second 2 ) , etc . 

Behavior data can be selected out of behavior 
data in the behavior database 45 in sequence from the 
top or at random. Here, behavior data are assumed to 
be selected in sequence from the top. First, 
acceleration a = -1 ( meter/ second 2 ) is selected. When 
this behavior is executed, the robot moves more 
slowly or backward. Since this is contrary to the 
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instruction "Speed up 1 11 issued by the user, the user 
inputs NG. Then, this behavior is cancelled, and the 
process restores to the state before starting a new 
behavior or the operation stops. Then, a behavior 
different from the previous behavior, acceleration a 
= 0 (meter/second 2 ) is selected and executed. By this 
the speed becomes constant. Since this is also 
contrary to the issued instruction, the user inputs 
NG this time too. Then, a next behavior, acceleration 
a = 1 (meter/second 2 ) is selected and executed. Then, 
the speed gradually increases, and the user inputs 
OK. 

The okayed behavior, acceleration a « 1 
(meter /second 2 ) and the issued instruction "speed up!" 
are combined by the word provision unit 48, and using 
data such as "instruction: speed up! -> Behavior: 
acceleration a = 1 ( meter/ second 2 ) " or data name Wd, 
data "instruction: Wd -> behavior: acceleration a = 1 
( meter /second 2 ) " are created and stored in the 
behavior database with a word 49 . 

By the above-mentioned procedures an instruction 
"speed up!" and a correct behavior "acceleration a = 
1" (meter/second 2 )" can be combined. 

The process in the case where an instruction 
which is similar in meaning to, but different in 
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words from an instruction "speed up!", for example, 
an instruction "at a higher speed!" is issued, is 
described below. Currently, data "instruction: speed 
up! -> behavior: acceleration a = 1 (meter/second 2 )" 
are assumed to be stored in the behavior database 
with a word 49. When an instruction "at a higher 
speed!" is issued, a data comparison is performed. 
However, since there is no matching data, a behavior 
selection is performed. At this time, since the 
behavior selection and judgement starts from a = -1 
in order, and finally at a = 1 OK is obtained, the 
instruction "at a higher speed!" is also combined 
with a - 1, and data "instruction: at a higher speed! 
-> behavior: acceleration a = 1 (meter/second 2 )" are 
stored in the behavior database with a word 49. Thus, 
two sets of instruction data are stored in the 
behavior database with a word 49. 

On the other hand, when an instruction "turn!" 
is issued, if there is no behavior pattern relating 
to a rotation in the behavior database 45, no 
favorable judgement can be obtained, even if all 
behaviors are tried. In this case, since finally 
nothing is OK, even if all accelerations are tried, 
an inquiry is issued to a user through the inquiry 
unit 52. For example, such an inquiry as "There is no 
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appropriate behavior. Designate an appropriate 
behavior!" is issued. 

In this case, this inquiry can be handled by 
adding such a behavior pattern as rotation speed - 
lrad/second to the behavior database 45. 

Alternatively, speed data at the time of 
behavior execution can be obtained in advance by a 
sensor 50, and the change patterns can be stored 
together with the behavior, or only patterns of 
sensor data can be stored together with the 
instruction in the behavior database with a word 49. 
In this case, since, when a certain instruction is 
issued, a user can understand how the state of the 
apparatus changes by the presentation of the pattern 
of the sensor data of a behavior matching with the 
instruction, a user can freely issue an instruction. 

If instructions are issued in sequence in this 
way, a variety of relationships between words and 
behaviors regarding the increase and decrease of 
speed can be built up, and thereby human instructions 
in a variety of patterns can be handled. 

Each unit executes the following processes. 

Fig. 22 is a flowchart of the process of the 
comparator unit 43 . 

When there are no data with a word which are not 
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yet compared with an input word in the behavior 
database with a word 49, that is, when as a result of 
the comparison in the succeeding steps S63 and S64 
(step S61), data with a word corresponding to the 
input word are not obtained, the comparator unit 43 
activates the behavior selector unit 44, and requests 
a behavior selection process (step S62). If there are 
not-yet-compared data with a word in the database 
with a word 49, the comparator unit 43 selects a not- 
yet-compared word (instruction) from the database 
with a word 49 (step S63), and checks whether or not 
the word matches with the input word (step S64). If 
the words match with each other, the comparator unit 
43 activates the behavior execution unit 46 to 
request a behavior execution process (step S65). If 
the words do not match with each other, the flow 
returns to step S61 and the comparison is repeated. 

Fig. 23 is a flowchart of the process of the 
behavior selector unit 44. 

The behavior selector unit 44 checks whether or 
not the behavior selection frequency so far reaches 
a predetermined frequency (step S71), and if the 
frequency reaches the predetermined frequency, the 
flow proceeds to step S73. If not, the behavior 
selector unit 44 checks whether or not there are 



67 

unexecuted behavior data in the behavior database 45 
(step S72), and if not, the flow proceeds to step 
S73. If there are unexecuted behavior data in the 
behavior database 45, the flow proceeds to step S76. 

When the selection frequency reaches the 
predetermined frequency or when all behaviors in the 
behavior database 45 are executed, the behavior 
selector unit 44 judges whether or not an inquiry 
should be issued (step S73 ) . If an inquiry is issued, 
the behavior selector unit 44 activates the inquiry 
unit 52, and performs an inquiry process (step S74), 
while, if not, the behavior selector unit 44 
terminates the process. 

On the other hand, if the selection frequency 
does not reach the predetermined frequency, and there 
are still unexecuted behavior data in the behavior 
database 45, the behavior selector unit 44 selects an 
unexecuted behavior (step S76), and turns ON the 
executed flag of the selected behavior (step S77). 
Then, the behavior selector unit 44 increments the 
selection frequency by 1 (step S78), activates the 
behavior execution unit 46 , and requests the 
execution of the selected behavior (step S79). 

Fig. 24 is a flowchart of the process of the 
judgement unit 47. 



• * 
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The judgement unit 47 checks whether or not 
there are criteria in the criteria database 51 (step 
S81). If there are criteria, it extracts the criteria 
from the criteria database 51 and verifies input data 
5 according to the criteria (step S82). If not, the 
judgement unit 47 waits for a judgement input from 
_ the judgement input unit 53 (step S83 ) . 

yj As a result of the comparison and judgement 

^ input, if the judgement is OK (step S84), the 

Jfj 10 judgement unit 47 activates the word provision unit 

Hi 48 and requests a word provision process (step S85). 

7*~ If not, the judgement unit 47 activates the behavior 

selector unit 44 and requests a behavior selection 
□ process (step S86). 

jg 15 As described above, according to the present 

03 

invention, since for the process of sensor data both 
a mechanism for classifying and storing input data 
from sensors and a mechanism for providing these 
classified data groups with appropriate words are 

20 provided, the status and state of the current 
location from the obtained sensor data can be 
correctly judged. Since enormous amounts of data are 
efficiently simplified and stored, and can be 
extracted and compared, if a status judgement is 

25 needed, better technology for the use of data can 
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also be provided. 

According to the present invention, a variety of 
instructions from users can be used, because a 
mechanism for selecting and executing behavior 
5 patterns appropriate for the instructions spoken by 
the users and a mechanism for combining and storing 
those behavior patterns and the instructions are 
provided. In addition, since an interface easy for 
users to understand is provided, a device operation 
10 apparatus with a better language process interface 
can be configured. 



